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Abstract 

The I LI AS learning management system (LMS) was evaluated, following its favourable rating in an 
independent evaluation study of open source software (OSS) products. The current review found 
I LI AS to have numerous features of value to distance education (DE) students and teachers, as 
well as problems for consideration in the system's ongoing development. The current findings 
were compared with DE students' reactions to a similar LMS product, ATutor, also rated highly in 
the independent OSS evaluation. In comparing an ATutor course website with a simple HTML- 
based version of the same site, the ten students voted unanimously to retain the simpler site. This 
result is consistent with previous evaluation findings in the current series of reports, and indicates 
that increasing integration of product features does not necessarily improve a product's ease of 
use or educational effectiveness. 


Introduction 

As the current series of reports has indicated, the number of online learning management systems 
(LMS) has proliferated, and at least a hundred are currently available. Leslie (2004) has identified 
over 50 such systems based on open source software (OSS) alone. In evaluating 35 OSS systems, 
Farrell (2003) has rated ATutor and I LIAS as the best, with Moodle close behind. The current 
evaluation studies have so far evaluated Moodle (Report #25) and ATutor (Report #37), rating 
both products favourably with respect to their potential value in distance education (DE). The 
current report completes this trilogy of reviews by examining the DE merits of I lias. In deciding 
the criteria for its review, the evaluation team reviewed the LMS selection standards used by (1) 
the American Society for Training & Development (ASTD: see previous reports in this series); (2) 
the Commonwealth of Learning evaluation study by Farrell (2003); and (3) the software 
comparison service, Edutools.info (2003). The criteria considered the most appropriate were 
combined and some new ones added. The final set of criteria is considered to represent the 
judgments that administrators, facilitators, designers and end users need to make in determining 
the value and usefulness of an LMS in DE. I n addition, previous online opinions of I LIAS were 
considered in the light of the version currently reviewed (v. 3.2.O.). Following the I LI AS review, 
the report also examines the trends that have evolved in the review series so far, with respect to 
DE students' reactions to complex, fully- featured LMS platforms, in comparison with those 
described in Report #8 as "lean and mean". In keeping with the convention adopted earlier in this 
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series (Report #25), the LMS acronym is adopted in order to avoid the confusion of terms 
currentiy arising in the educational software literature. 


Evaluation of Ilias 

I LI AS is a free, open source (OS) Web-based learning management system (LMS) developed at 
the University of Cologne (2004). Originally released in J uly 2001, with early versions dating from 
mid-2000, ILIAS stands for Integrated Learning, Information and Co-Operative Work System. It 
has been designed as a coliaborative, hypermedia working/ learning environment, fiexibie for use 
by teachers with different pedagogical approaches, and allowing the reuse and exchange of 
materials based on course-sharing standards (e.g., I MS, The Aviation I ndustry CBT [Computer- 
Based Training] Committee (AlCC) and ARIADNE Foundation for the European Knowledge Pool). 

As the previous report in the series (Report # 42) has indicated, all OSS is by definition a work in 
progress, constantiy enhanced and modified as a result of user input and coliaboration. ILIAS is 
no exception. Its website currentiy reports its use on over 100 servers, mainiy at Universities in 
Europe and Asia. In the OSS sharing spirit, therefore, the current evaiuation team contacted the 
four listed North American users with inquiries about their ILIAS experience (though none 
responded). 

Particular aspects of ILIAS that support the moderate constructivism phiiosophy are: 

• Team based authoring: I LIAS allows for the collaborative development of course materials. 

• Personal annotations: ILIAS offers a personal context- related annotation function that permits 
a iearner to add summaries, comments and annotations to text elements, graphics or images. 
These annotations can be retrieved at any time within the original context, printed out in a 
structured form, and exchanged with members of a learning group. 

• Personal bookmark folder: learners can build and administer content- related resources 
available from the Web. 

• Learning as a seif-directed process: users can choose the learning materiais to review, when 
to review them, and the sequence of materials. The materials are accessed from a central 
database which only displays the content relevant for the learner's current concerns and 
focus. A search engine aliows the user to seek specific content within the materials. 

• Learning is a social process: facilitators and learners can create groups of users and define 
access privileges. 

• Text elements are assumed to achieve the best educational effect for differently targeted 
learning groups: the text elements form the 'backbone' of an I LIAS course. Other elements 
(e.g., animations, interactive elements, co-operative elements, graphic, audio, and video) 
support this basic structure. Once the eiements have deveioped and integrated into an ILIAS- 
based course, the author is free to associate and link them based on a specific pedagogical 
approach. 

ILIAS is a cross-browser system, requiring Internet Explorer 6.0, Mozilla 1.4, Safari 1.0, and/ or 
Lynx 2.8.4 (or higher), and runs with JavaScript disabled. The system is frames-based, cannot 
support text-only navigation, and is not usable with iarge fonts. It requires MySOL 4.0.16 or 
higher. A digest of the product's features is as foliows: 

• Text discussions (viewable in threads and associated with course content) 
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• Integrated real-time text chat (java-based Babylon) 

• File transfer available between users 

• Small-group discussion environments (created by teachers and students) 

• Each group can have its own discussion forum, chat room, and file exchange 

• Web-based mailing system with usermail, groupmail, and integrated send mail functions 

• Personal student homepage (desktop) linking to courses, groups, bookmarks, and email 

• Students can attach notes to displays, and can search and compile them for the course as a 
whole 

• Course content is downloadable for offline work and local printing/ storage 

• A printable study guide can be generated 

• Polls can be administered with online and print-based statistical reports 

• Teachers can create self-assessment tests (multiple-choice and cloze) with automatic scoring 

• Teachers can control a student's progress through course modules based on test results 

• Students can use specified criteria to compile a collection of exercises from an item test bank 

• Student exercises can yield a feedback report of correct and incorrect responses 

• Administrators can set courses to be publicly accessible or can restrict access by ID and 
password (author, learner, administrator, and guest levels) 

• Specific tasks can be assigned at each access level 

• Users can maintain their own passwords 

• Detailed reports tracking access to course materials, though not for individual users 

• Integration with institutional IT architectures via LDAP/ RADIUS authentication 

• Third party hosted services are available (in German only) 

• No whiteboard (scheduled for next version) 

I LI AS currently suffers from limitations common to many OSS products. These include: lack of 
interoperability and integration; lack of support for audio/ video-conferencing, time- zone or date 
localization; poor accessibility standards; and an unclear estimate of the true cost of ownership. 
Currently, many OS LMS platforms lack integration with the applications typically used for 
registration and employee management in universities, colleges, and industry (Bailetti, 2004). The 
same writer also notes that OSS systems tend to lack good user, developer and administration 
documentation; and this is true of I LIAS. Although the software contains a 13-language 
translation facility, up-to-date documentation for it is available only in German; non-German 
users have little chance of influencing the product's development via participation in the various 
help forums. (On the other hand, Farrell (2003) notes a growing following for the product in 
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China.) While I LI AS is classified as freeware, its use is not actually cost-free. The time spent in 
installing and maintaining the system must be added to the costs of the hardware, technical 
support and training; and the current evaluators (both experienced server administrators) found 
the logistical problems of I LIAS installation to be considerable. They spent several days trying to 
install and configure it on Linux, in conjunction with its dozen or more required third party 
applications. Poor English documentation contained conflicting instructions. As a last resort, the 
messages in the German-language I LI AS user forums were scoured, until an English description of 
a Windows-based installation alternative was found. With this, I LI AS was working on a 
Windows/2000 server within 30 minutes. Despite its many current shortcomings, it has an 
admirable range of features, pedagogical integrity, and good potential for DE users. OSS 
applications such as this have come a long way within the three years of the current evaluation 
series, and they are continuing to advance quickly. 

Previous reports in this series have indicated, however, that fully-featured online tools do not 
necessarily find favour with DE users. Students can find them difficult to master, and DE teachers 
do not necessarily share a need for the many features they offer. In addition, their integration 
with existing institutional systems can be expensive, and the overall costs of proprietary systems 
especially, may be excessive in the relation to the actual uses to which they are subjected. In 
Report #8 in this series, a direct comparison was made between DE students reactions to a fully- 
featured conferencing product and a simple 'bare-bones' alternative. Unexpectedly, a majority of 
students preferred the simple application, and regarded a transfer to the more complex option as 
not worth the effort. In the two years since that study, online collaborative tools have converged 
within ambitious integrated products such as WebCT Vista, and in OSS applications (e.g., ATutor, 

I LIAS, and Moodle). Having now established that each of these products has a wide range of 
features of use to DE teachers and students, the next section of this report replicates the earlier 
study, by asking DE students to compare the comprehensive open-source ATutor product with the 
more limited HTML-based site of their current online course. 


Comparison of Course Website Formats 

Ten students in a graduate Masters in Distance Education (MDE) course at Athabasca University - 
Canada's Open University, were asked to compare the simple, home-grown, frames-based format 
of their current course website (see Figure 1) with that of the OSS LMS ATutor. The content of the 
course site was replicated in ATutor (see Figure 2), using the free trial and hosting facilities of the 
product's website. ATutor was selected for this purpose in view of its top rating by Farrell (2003), 
and its favourable evaluation in Report #37 of this series. The current study fulfilled a promise 
stated in that report, that ATutor would be adopted for the MDE course depending on feedback 
from its current students as a whole. 

Figure 1. The frames-based format of an online Master's in Distance Education course website 


http://www,irrodl.org/index,php/irrodl/rt/printerF riendly/213/296 


7/3/2008 



Hotrum 


Page 5 of 8 


•f IHitatwr F4«r*(ke« M2 


CMnec«y«r pa^e Im A t ki b a K a U cm t ««y*i Matur ^ Cuiamet SJiieaaoH 662: 

<MDOE4<2) TW <«r»* <o¥w dw racg^ ac^KriM w< >x4 o^t uk 4 fcf if M i 
t4»cakc*a] <:<rf»iacf U W ff fcanra totewart rrjluAo« »Mef, aoi wd pr«v»4a «» 4fc» t «n(h a* >» 

f«iMab &*■ « A* I>aur^:te^*kdi Afvtr>« tn ami Cimaae* Laa ^m m g TW <««>t wd 

baaed « part «*tfa4>ir«fikr CDF I cetab««rr i»«a lOr AaC celatei Ote re«di «f Ae cootk'i rrdaaAo* RuAea 

The f Mtee «iU nee« m a refolat Wacu vm Htetwt m4m. m4 wfl vcAa* fppckrcoaw. paced mA*v4«^ actRAei 
TVeae wd n'^mt dl c««r»e ewnben le uae icAwarr pec^ucti ockalMgAw oakie m^y freeware, Yakoo Htmtntf* 
WbrapMhare dcwateadei andauialrd rack p«r<* «f aoAwsv. pSrair c-aaa^ the duwucAe wiAp»« racae. tcAwair 

MBT. a»i At n> (trtbniwt) «f pm a«o«ai 

MCDE6b2 wd alto c«Mder Ae itnf yr of wf AAweW »nl«f ctiima— cAot leclnuw » partctiar MuAoaa 

• To arwl ameatace to thr mWikW weotAietxe, pJraar «ie rrfd* «-«Md 

• To Mtl a brief neifa(f •• Ae tMctir. tf ifi— ( ■ leief itipo«» oclp. we Ac AM-eMfrafe boa of 
Y o A oc U*t»tm§tr FAeaelrvKtcea nelwe. he wll #ee pea a top-prK«ay. tutata reply* 

• To acwl a ^tetbos or cowaeat AR w aelJ be ofb w e ft to the oAee CMrae oteoArr*. pleaM ir Ae a p«Mc 
pothBC e Ar affeopeule cowte led Atcvcexie 

• To afraa^ an auAo Aacuiiko* wAi Ae iM«iK«4e. «■( Katboo hUamnftr. check tf be m aeatfAk bp 

* ^itftnd OMCtaAe. CiAMt Ar. aeod fan a few akeraabwe 4R4 I^Am> bp e-aatf 

X bof« po« wd fibl MDDEb42 ea^tfbk mi peoAxtirc' Jfl! 


Figure 2. A sample configuration of ATutor (same course as Figure 1) 



A representative selection of the students' responses to the two course platforms is as follows. 


Student A: 

• I am very impressed with ATutor. Based on the available tools and the initial appearance, it 
appears to have all of the features of WebCT . . . plus much more customization. 

• For the most part, navigation seems very intuitive, particularly at the higher levels. Once I 
get into the lower menu levels such as at this step of posting a message, the green and red 
circles confused me . . . 

• I did not like the built-in email feature in WebCT, nor in this program either. I prefer using my 
regular email program. Flaving this feature just means I have another email box to check. 
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• The chat is a nice tool, but I would rather use something like Yahoo Messenger. This chat only 
works if you are logged into the course. Yahoo works whether you are in the course, another 
course, or doing something completely unrelated to the course. It is much easier to stay in 
contact with individuals with Yahoo rather than using a built-in tool such as this that has only 
one specific use. 


Student B: 

• There are only so many things that can be done with a text based conferencing format. 

• I do not like to use an additional email . . . just adds to my daily chores. 

• I don't personally like the "I'm online" feature ... we can just be so over-solicited and 
sometimes it's good to just go about one's business in peace. 

• I love variety, but I want simple, consistent, predictable access to my studies. Getting to 
them, I already have enough obstacles. 


Student C: 

• Too many tools makes for a cluttered toolbox with more time spent sorting than working. 

• With so many tools and so many changes, one (cannot) get comfortable with an application 
before having to move on to the next version. This is an example of productivity paradox. 

• My initial impression of ATutor was that it was busy. Icons are nice, but they take up screen 
real estate. 

• I 've never been a big fan of help files. I prefer to jump in and get my hands dirty. A truly 
intuitive application makes limited use of help files. 


Student D: 

• ATutor interface is extremely cluttered, although I like the icon/ text option - meets 
accessibility needs. At least as a user you can use preferences - maybe too much opportunity 
to change. 

• (The many features are) like the result of a progressive build - someone wants something 
added, so it's added. 

• I find the navigation is not intuitive, and the help files too lengthy and require scrolling - no 
search feature. I think screen clutter impacts upon my comfort with the navigation. 

• I like the embedded links to our MDE course, but they do clutter the screen. 


http://www,irrodl.org/index,php/irrodl/rt/printerF riendly/213/296 


7/3/2008 



Hotrum 


Page 7 of 8 


Student E: 

• I have been reflecting on how my rural, barely computer literate students will react to this 
software. I believe I would need to send detailed written instructions on how to logon and 
navigate. Online help is distracting and hard to use if you are uncomfortable in this 
environment. 


Student F: 

• In terms of interaction usability, at first I didn't find it that intuitive . . . wasn't sure what 
some of the circle icon meant. However, the user interface was for the most part fairly 
predictable. 

• Not impressed with some of the "bells and whistles" like the internal email . . . just one more 
thing to worry about. 

• I do like some of the site tools. Changing the preferences made the overall appearance more 
palatable for me. Also, the search and site- map features made navigating more purposeful. 


Despite ATutor's clear technical superiority in comparison with the existing HTML-based course 
site, the majority of students voted to retain the existing frames-based site shown in Figure 1. 
None voted to move to the ATutor platform. This result was disappointing to the course instructor, 
who had hoped for student feedback justifying a move to the superior support that would be 
available for ATutor in the University. A common theme in the students' responses is their 
reaction to the ATutor display as too "cluttered/ busy" with too many "obstacles/ bells and 
whistles." These responses are essentially the same as those reported in the similar study 
reported earlier in the series (Report #8). The clearest explanation for this preference for a simple 
course delivery format was provided in the present study by a student who stated: "I see 
advantages to both, but nothing compelling enough to make a change." 


Conclusions 

The recurring finding of these studies brings into question the wisdom of providing DE students 
with ever more complex online learning systems; and it suggests the need to balance 
comprehensiveness of product features against clarity of usage. In the absence of unequivocal 
justification for moving to a new learning platform, both teachers and students are apt to select 
"the devil they know". If new systems have definite advantages, these should be carefully pointed 
out to them. It is inevitable that the developers of online software (OSS as well as proprietary) 
will continually offer "bigger and better mousetraps" on the assumption that the more features 
their products contain, the more competitive they will be. OSS as well as proprietary software 
developers are moving in this direction; and the increasing comprehensiveness of LMS software 
such as I LI AS and ATutor is evident from the current product reviews and those provided by 
Farrell (2003) and edutools.info (2003). Reviews based solely on the volume of a product's 
features, however, do not necessarily predict users' reactions to the product or its educational 
effectiveness; and it is clear that simple preferences for proprietary or OSS approaches do not 
predict the educational merits of a product either. DE students' reactions to the highly rated I LI AS 
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and ATutor products, as expressed in the present report, suggest that too much product 
integration can be counter-productive in the oniine iearning process, and that teachers and 
students may both benefit from a simpie, moduiar-based approach that aliows them to expand 
their oniine toois and skilis as and when they need to do so. 
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The next report in the series will continue the current discussion of attitudes to LMS methods. 

N.B. Owing to the speed with which Web addresses are changed, the online 
references cited in this report may be outdated. They are availabie, together with 
updates to the current report, at the Athabasca University software evaiuation site: 
http://cde.athabascau.ca/softevai/. Italicized product names in this report can be 
assumed to be registered industriai or trademarks. 
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